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1.0  SUMMARY 


As  combat  operations  draw  down  and  budgets  become  tight,  military  decision  makers  are  faced 
with  difficult  decisions  regarding  how  to  allocate  resources.  Decision  makers  seek  quantifiable 
evidence  to  justify  any  investment,  and  investing  in  training  is  no  exception.  Training  system 
return  on  training  investment  (ROTI)  is  rarely  calculated  even  though  it  can  reveal  significant 
insight  regarding  the  cost,  schedule  and  training  performance  capabilities  of  these  systems. 
Optimizing  ROTI  can  yield  significant  savings  in  costs  associated  with  reduction  in  training  time 
and  a  decrease  in  instructor  resources  or  compression  of  the  time  necessary  to  produce  levels  of 
expertise  (Cohn  and  Fletcher,  2010).  Part  of  the  reason  these  types  of  analyses  are  not  conducted 
is  their  resource  intensiveness,  lack  of  standardized  methodologies  and  limitations  on  data  to 
make  calculations.  What  is  needed  is  a  standardized  ROTI  method  and  tool  which  is  accessible, 
easy  to  use,  and  generalizable  across  domains.  Under  the  RITE-MT  effort,  the  Evaluation  of 
Value  Added  in  Learning  Systems-RITE-MT  (EVALS-R)  tool,  a  decision  support  tool  for  ROTI 
analysis,  was  developed  to  address  this  gap  by  utilizing  a  Training  Needs  Analysis  (TNA)-based 
approach  which  incorporates  the  quantification  of  performance  (i.e.,  training  effectiveness)  gains 
with  respect  to  target  training  objectives,  and  lifecycle  cost  and  schedule  impacts.  EVALS-R  1) 
provides  an  analysis  of  a  training  system’s  ability  to  meet  training  goals,  within  the  budgetary 
and  schedule  constraints  set  forth  by  the  organization,  and  2)  facilitates  return  on  investment 
tradeoff  analyses  both  within  a  system  (e.g.,  which  specifications  justify  their  cost)  and  across 
systems  (e.g.,  which  systems  provide  the  most  “bang  for  the  buck”),  resulting  in  increased 
organizational  training  and  readiness.  To  accomplish  this,  the  RITE-MT  effort  completed  the 
following  objectives: 

Objective  1:  Plan  EVALS-R  Effectiveness  Evaluation 

Objective  2:  Define  EVALS-R  Requirements  and  Specifications 

Objective  3:  Conduct  Detailed  Design 

Objective  4:  Develop  Prototype  Software 

Objective  5:  Conduct  Verification  and  Validation 

Objective  6:  Design-Code-Test  Version  2.0 

Objective  7:  Perform  EVALS-R  Effectiveness  Evaluation 

Objective  8:  Develop  Commercial  Production  and  Distribution  Plan 

2.0  INTRODUCTION 

The  Evaluation  of  Value  Added  in  Learning  Systems-RITE-MT  (EVALS-R)  was  developed 
under  this  effort  using  an  agile  software  development  process  to  expedite  the  design- 
development-evaluation  cycle  and  integrate  greater  flexibility  into  the  direction  of  the  systems 
design.  This  process  allowed  for  iterative  functional  testing,  redesign  requirements  definition, 
and  implementation  of  prototype  development  changes  within  each  version  of  the  system,  and 
quicker  integration  of  feedback  and  adjustments  to  the  system’s  design.  Though  user  needs 
identification,  requirements  development,  and  detailed  design  were  largely  completed  before 
prototype  development  began,  the  agile  software  development  process  employed  allowed 
revisions  to  the  design  as  needs  were  identified.  This  increased  the  efficiency  of  development  by 
allowing  for  bugs  to  be  identified  and  mitigated  quicker  (while  they  still  have  minimal  impact  on 
the  software),  allowing  for  extra  functionality  to  be  developed  in  each  version  in  the  time  that 
would  normally  be  spent  fixing  bugs.  Initial  user  needs  identification  and  preliminary  testing 
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with  interface  designs  were  done  with  subject  matter  experts  (SMEs)  from  the  Nicholson  Center 
at  Florida  Hospital  and  Project  Manager  Training  Devices  (PMTRADE).  Additional  redesign 
requirements  were  determined  from  usability  testing  of  different  versions  of  the  system  with 
SMEs  from  Project  Manager  Training  Systems  (PM  TRASYS),  other  acquisition  professionals, 
and  human  factors  engineers. 

3.0  METHODS,  ASSUMPTIONS  AND  PROCEDURES 

During  the  period  of  performance,  Objectives  1-8  were  completed,  beginning  with  the 
development  of  the  effectiveness  evaluation  plan  and  submission  to  the  Institutional  Review 
Board  (IRB)  for  approval,  followed  by  identification  of  user  needs  during  subject  matter  expert 
(SME)  interviews  with  SMEs  from  the  Nicholson  Center  at  Florida  Hospital  and  from 
PMTRADE  to  revise  the  concept  designed  in  Phase  I.  Then  requirements  and  specifications  were 
developed  for  the  EVALS-R  software  tool,  including  use  cases  of  how  the  tool  will  be  used. 
These  requirements  led  to  the  creation  of  a  detailed  design  of  the  system,  including  the  overall 
architecture,  information  flow,  and  graphical  user  interfaces  (GUI).  An  agile  software 
development  process  was  employed  while  developing  each  version  of  the  software  to  allow  for 
the  flexibility  to  iteratively  test  and  implement  changes,  increasing  the  efficiency  of  development 
and  increasing  the  amount  of  functionality  able  to  be  included.  Usability  testing  of  the  different 
versions  with  different  potential  end  users,  including  an  acquisitions  professional  from  PM 
TRASYS,  resulted  in  redesign  requirements  that  were  incorporated  into  the  following  versions, 
resulting  in  a  final  version  with  minimal  usability  issues.  Finally  the  commercial  production  and 
distribution  plan  was  developed. 

3.1  Objective  1.  Plan  RITE-MT  Effectiveness  Evaluation 

The  protocol  for  the  effectiveness  evaluation  was  developed  and  submitted  for  IRB  review. 
Approval  was  received  on  September  18,  2013. 

3.2  Objective  2.  Define  RITE-MT  Requirements  and  Specifications 

The  system  requirements  for  the  RITE-MT  (EVALS-R)  system  were  developed  based  on  the 
user  needs  gathered.  The  requirements  were  then  reviewed  by  the  software  development  team, 
and  revisions  were  made  in  the  final  version  based  on  their  feedback. 

3.3  Objective  3.  Conduct  Detailed  Design 

The  design  of  both  information  flow  and  the  overall  the  architecture  of  the  system  as  well  as  the 
design  of  all  of  the  Graphical  User  Interfaces  (GUIs)  was  completed  based  on  the  use  case 
designs  and  interactively  improved  through  redesigns  following  feedback  or  evaluations.  These 
designs  were  shown  to  potential  end  users  (both  government  acquisitions  SMEs  and  a 
commercial  medical  training  SME)  and  feedback  was  incorporated  into  revised  designs  before 
development  of  the  system  prototype  began. 


3.4  Objective  4.  Develop  Prototype  Software 

With  a  better  understanding  of  the  scope  of  the  effort  based  on  the  conceptual  refinements,  a 
software  development  plan  was  put  together  to  ensure  that  the  software  development  goals 
would  be  obtained  within  the  schedule  and  budgetary  constraints  of  the  Phase  II  effort.  An  agile 


7 


software  development  process  was  used  to  develop  the  components  of  the  EVALS-R  system 
which  resulted  in  three  comprehensive  versions  being  developed  and  improved  upon  throughout 
Phase  II. 

3.5  Objective  5.  Conduct  Verification  and  Validation  (V&V) 

Testing  of  the  software  application  was  conducted  throughout  the  agile  development  process. 
Performance  was  tested  against  functionality  defined  in  the  requirements  and  use  case  documents 
to  ensure  the  system  contained  the  desired  capabilities. 

3.6  Objective  6.  Design- Code-Test  Version 

Testing  ensured  the  delivered  system  contain  no  major  errors  or  undesired  effects  (i.e.,  “bugs”). 
This  process  was  conducted  throughout  the  agile  development  process. 


3.7  Objective  7.  Perform  RITE-MT  Effectiveness  Evaluation 

The  effectiveness  evaluation  of  the  EVALS-R  system  focused  on  ensuring  adequate  end-user 
interaction  and  task  support  (i.e.,  usability)  as  per  direction  of  the  project’s  sponsor  (COR  in  a 
personal  communication).  This  evaluation  was  conducted  with  10  individuals  who  represented 
potential  end  users  (e.g.,  two  SME  with  10  and  35  years  of  acquisitions  experience  respectively, 
and  8  others  with  various  years  of  human-systems  integration  experience).  Usability  findings 
were  mostly  positive,  with  a  few  issues  that  were  addressed  via  redesign  of  some  of  the 
interfaces. 

3.8  Objective  8.  Develop  Commercial  Product  and  Distribution  Plan 

Preparation  of  regression  testing  of  the  EVALS-R  system  was  conducted  to  ensure  the  system’s 
readiness  for  release.  A  user  manual  was  developed  and  can  be  found  in  the  help  section  of  the 
software.  The  system  was  also  demonstrated  to  multiple  potential  customers,  with  the  most 
interest  from  PM  TRASYS,  who  provided  a  letter  of  support  for  a  Rapid  Innovation  Fund  white 
paper  that  was  submitted  but  not  accepted.  Through  evaluation  and  reviews  with  PM  TRASYS, 
additional  system  capabilities  /  requirements  were  identified  which  would  be  outside  the  scope  of 
Phase  II  efforts  Therefore,  alternative  potential  funding  opportunities  are  being  sought  to  tailor 
the  EVALS-R  system  to  their  needs. 

4.0  RESULTS  AND  DISCUSSION 

4.1  Evaluation  of  Value  Added  In  Learning  Systems-RITE-MT  (EVALS-R)  Tool 

The  primary  result  of  this  Phase  II  effort  was  the  development  and  validation  of  the  EVALS-R 
system.  This  was  the  result  of  the  evolution  of  the  Phase  I  design  which  incorporated  a  review  of 
the  return  on  training  investment  literature,  interviews  with  SMEs  and  prototype  designs.  During 
Phase  II,  additional  front-end  analysis  was  conducted  to  expand  on  the  Phase  I  findings  primarily 
by  seeking  input  from  potential  transition  customers  and  SMEs  from  both  private  and 
government  sources  (e.g.,  PMTRADE,  PM  TRASYS,  Nicholson  Center  for  Surgical 
Advancement  at  Florida  Hospital).  This  insight  was  integrated  into  the  conceptual  and  detailed 
design  of  the  EVALS-R  system,  which  was  then  integrated  into  a  development  plan. 


EVALS-R  offers  a  number  of  key  benefits  to  the  training  system  acquisition  community.  Up 
until  now,  the  use  of  ROTI  analysis  has  been  limited  and  non-standardized  using  decentralized 
processes  mostly  handled  through  multiple  spreadsheets.  This  has  presented  challenges  in  reuse 
of  data  and  general  usability  of  the  process,  and  has  resulted  in  suboptimal  use  of  resources  or 
acquisition  recommendations.  EVALS-R  addresses  these  challenges  by  providing: 

•  ...a  tool  where  standardized  ROTI  analyses  may  be  conducted  and  stored. 

•  ...a  ROTI  methodology  that  translates  cost,  schedule,  and  performance  data  into 
actionable  data  and  score. 

•  ...a  tool  where  training  domain  (training  requirements)  and  training  system  data  may  be 
stored  for  reuse  at  later  times. 

•  ...a  network  capable  tool  which  can  support  multiple  distributed  users  if  so  desired 
(standalone  isolated  setup  within  a  local  network  is  also  supported). 

•  ...sensory  analysis-based  diagnostic  capabilities  to  understand  the  training  capabilities  of 
the  systems  under  review. 

•  ...a  baseline  system  from  which  additional  analytical  capabilities  and  customization  may 
be  introduced. 

The  Phase  II  conceptual  design  of  the  EVALS-R  system  is  illustrated  in  Figure  1  below.  This 
conceptual  design  highlights  the  training-centered  approach  utilized  within  the  EVALS-R  system 
where  the  analysis  on  training  investment  is  conducted  with  regards  to  the  training  capabilities  of 
the  evaluated  system.  The  EVALS-R  system  is  organized  around  four  key  main  components  of 
the  system:  1)  Training  Domains,  2)  Training  Systems,  3)  Evaluations,  and  4)  ROTI  Analyses. 

EVALS-R 


ROTI  Scores 


I 

Tradespace 


Figure  1.  EVALS-R  Conceptual  Design 

The  Training  Domains  component  (Figure  2)  is  a  persistent  library  of  operational  training 
domains  intended  to  grow  with  use  of  the  system.  Here  users,  usually  operational  domain  SMEs, 
are  guided  through  a  characterization  method  to  decompose  training  tasks  and  identify 
operational  cue  requirements  (e.g.,  fidelity  cue  requirements). 
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Edit  Domain 


Task  Analysis  Edit 

Complete  a  task  analysis  by  setting  up  levels,  and  adding  measures  and  cues  Edit 


Q  Setup  Levels 


o 


1  Tactical  Combat  Casualty  Care 

1.1  Care  Under  Fire 

1.2  Tactical  Field  Care 

1.2.1  Assessment  consists  of  airway,  breathing,  and  circulation 


1.2.2  Airway 

1.2.3  Breathing 

1.2.4  Bleeding 


1.2.5  Intravenous  Access 

1.2.6  Fluids 


1.3  CASEVAC  Care 


1  Tactical  Combat  Casualty  Care 


Name 

Tactical  Combat  Casualty  Care 

Description 

Enter  Description 

Task  Criticality 

Non-Critical 


Add  New  Task/Subtask 


Figure  2.  Training  Domain  Screen 

The  Training  Systems  component  (Figure  3)  is  a  persistent  library  of  training  systems  that  have 
been  analysed  or  may  be  going  through  analysis.  This  component  allows  users  to  define  training 
system  characteristics  such  as  cost,  schedule,  and  system  specification  data  as  well  as  house 
historical  training  evaluation  data. 
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0  Training  System 


,ll 


Adult  ALS  Trainer  Edit 

This  system  provides  mono  directional  speakers  with  the  ability  to  create  pneumothroax  condition,  move  arms  and 
legs,  simulate  distorted  airways,  simulate  pulse,  bp  monitor,  and  produce  realistic  blood  Edit 


Delete  Training  System 


Variables 

Cost,  schedule,  and  performance  data  about  the  training  system. 


Cost 

It 

Schedule 

Variable  Type 

Purchase  Cost 

3535 

Dollars 

Cost  to  Acquire 

Cost  to  Deploy  System 

1000 

Dollars 

Cost  to  Acquire 

Delivery/Transport  Cost 

700 

Dollars 

Cost  to  Acquire 

Setup  Cost 

100 

Dollars 

Cost  to  Acquire 

Equipment  Cost 

0 

Dollars 

Cost  to  Acquire 

Figure  3.  Training  System  Screen  with  Subset  of  Variables 

Together  data  from  both  of  these  components  is  utilized  to  conduct  a  predictive  training 
evaluation  in  the  Fidelity  Evaluations  component  (Figure  4),  based  on  the  training  system’s 
ability  to  support  the  identified  required  fidelity  cues.  This  process  may  also  be  enhanced  with 
empirical  data  with  additional  system  capabilities  beyond  this  Phase  II  effort. 


EVALS  -  R 


Setup 


Fidelity  Evaluation  Medical  Simulators 


Print  Report 
Export  Data 


50%  of  overall  tasks  supported,  50%  of  critical  tasks  supported,  0%  of  non-critical  tasks  supported. 


Task 

Criticality 

Task  Supported? 

Info 

Managing  Pneumothorax 

Critical 

Tension  Pneumothorax  Management 

Critical 

View  Info 

Back 


Figure  4.  Fidelity  Evaluation  Results  Screen 


FINISH 


11 


The  results  of  the  training  evaluation  plus  cost  and  schedule  data  for  a  system  under  evaluation 
may  then  be  fed  into  the  ROTI  Analysis  component  (Figure  5).  The  ROTI  analysis  component 
guides  the  user  through  defining  a  ROTI  model  which  considers  the  cost,  schedule,  and  training 
performance  variables  and  their  relative  importance.  The  user  may  choose  to  include  and  weight 
the  variables  as  best  fitting  the  acquisition  goals  of  the  organization.  Once  the  cost,  schedule  and 
training  performance  data  is  fed  into  the  model,  the  system  presents  the  ROTI  results  in  the 
Tradespace  output  for  review  by  the  user.  Within  this  output  the  user  may  choose  to  adjust 
parameters  in  the  model  to  visualize  the  results  (e.g.,  change  cost  or  throughput  values). 


EVALS  -  R  O  ttshbo.* 

E5  Trailing  DomMn  OT™*gSy*«n  S  E^. 

.k  ROTI  Analysis 

RITEMT 

Print  Report 

Download  CSV 

Highest  ROTI  Score 

This  is  the  system  that  best  fits  your  needs.  A  score  above  0.5  means  that  on  average, 
the  system  exceeded  the  Objective  values  set.  A  score  between  0  and  0.5  means  that  on 
average,  the  system  exceeded  the  Threshold  but  cfd  not  meet  the  Objective. 


Hospital  Simulator  DEMO 

V  0.51 

ROTI  SCORE 

ROTI  Score  Comparison 


Figure  5.  ROTI  Analysis  Results 


4.2  Effectiveness  Evaluation  (Usability  Testing  and  Redesign) 

Throughout  the  Phase  II  effort,  the  EVALS-R  system  underwent  usability  focused  evaluations 
involving  both  expert-based  reviews  (e.g.,  heuristic  evaluations  with  wireframes,  detailed  GUI 
designs,  or  software  GUIs)  and  empirical  user  testing  with  representative  end-users.  For  this, 
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measures  of  task  execution  effectiveness,  efficiency,  errors  and  satisfaction  were  used  to  assess 
for  potential  usability  concerns.  Through  this  iterative  process,  usability  concerns  were  identified 
and  addressed  as  needed  through  redesign. 

At  the  same  time,  summative  measures  of  usability  were  utilized  to  understand  the  quality  of  the 
design  as  development  progressed.  For  this  the  system  usability  scale  (SUS,  Brooke,  1996),  and 
the  Cooper-Harper  subject  workload  scale  (Cooper  and  Harper,  1969),  an  easy-to-use,  validated 
workload  scale  which  simplifies  the  workload  construct  into  a  single  measure,  and  a  measure  of 
task  utility  were  utilized.  In  total,  19  participants  took  part  in  these  series  of  evaluations  of  the  3 
major  versions  of  the  EVALS-R  system.  What  follows  is  a  summary  of  the  observed  results. 

Usability:  Participants  answered  a  list  of  questions  regarding  usability  using  the  Likert  scale  (1- 
Strongly  Disagree  and  5-Strongly  Agree);  grading  was  conducted  using  instructions  provided  in 
Brooke  (1996).  Table  1  summarizes  these  results  where  an  overall  increase  in  the  SUS  score  was 


observed  from  the  first  to  the  last  version  developed  in  Phase  II.  The  observed  SUS  score  of 
76.67  in  V3.0  corresponds  with  high  usability  (Bangor,  Kortum,  &  Miller,  2009;  scores  68  and 
above  usually  correlate  with  systems  that  have  high  usability). 

Table  1.  Subjective  Usability  Scale  Results 

V1.0 

(n=9) 

V2.0 

(n=6) 

V3.0 

(n=4) 

ANOVA 

Measure  M  SD 

M  SD 

M  SD 

F  P 

SUS  (higher  is  better)  60.00  15.46 

59.17  14.17 

76.67  5.20 

1.9  0.184 

Workload:  The  Cooper-Harper  scale  utilizes  two  questions  to  determine  the  observed  workload 
perceived  by  the  user  and  the  expected  or  desired  level  of  workload  and  a  10-point  scale  (0- 
Completely  undemanding,  very  relaxed  and  comfortable;  9  -  Completely  demanding).  The 
results  summarized  in  Table  2  show  little  variation  in  workload  across  the  different  versions. 
Such  variations  may  be  explained  by  the  changes  in  design  approach  and  incremental 
functionalities  built  across  the  three  versions. 


Table  2.  Workload  Results 


V1.0 

V2.0 

V3.0 

ANOVA 

(n= 

-.9) 

(n= 

=6) 

(n= 

=4) 

Measure 

M 

SD 

M 

SD 

M 

SD 

F 

P 

Workload 

0.24 

2.00 

1.67 

•  Observed  (0-9  response  scale) 

3.54 

1.30 

5.17 

1.00 

4.67 

2.89 

1.88 

0.186 

•  Expected 

3.30 

1.13 

3.17 

1.53 

3.00 

0.00 

0.08 

0.923 

Utility:  As  a  measure  of  Utility,  two  questions  using  a  5-point  Likert  scale  (1-strongly  disagree  to 
5-strongly  agree)  were  combined  into  a  Net  Utility  score.  This  was  a  composite  of  how  much 
value  users  believed  the  system  added  versus  how  much  complication  was  created  by  the  system. 
Table  3  summarizes  these  results:  the  Net  utility  score  for  the  last  version  developed  under  Phase 
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II  was  1.67  (sd:  1.15)  of  a  possible  range  of  -4  (where  a  system  creates  more  complications  than 
help)  to  4  (where  a  system  helps  more  than  creates  complications). 


Table  3.  Satisfaction  and  Utility  Results 


V1.0 

(n=9) 

V2.0 

(n=6) 

V3.0 

(n=4) 

ANOVA 

Measure 

M  SD 

M  SD 

M  SD 

F 

P 

Net  Utility  Value 

1.22  0.97 

1.00  1.55 

1.67  1.15 

0.3 

0.745 

•  System  helps  with  tasks  and 
responsibilities 

3.44  0.73 

3.00  1.26 

4.00  1.00 

1.08 

0.366 

•  System  complicates  tasks  and 
responsibilities 

2.22  0.83 

2.00  1.10 

2.33  0.58 

0.17 

0.845 

Together  these  results  highlight  the  value  added  by  the  EVALS-R  system  by  providing  useful 
value  with  relative  ease  of  use.  These  capabilities  will  serve  training  system  acquisition,  research 
professionals  and  subject  matter  experts  without  adding  significant  burden  in  its  use. 


4.3  Commercial  Production  and  Distribution  Plan 

The  EVALS-R  system  helps  training  system  acquisition  professionals  who  want  to  understand 
and  optimize  the  return  on  training  investment  of  systems  they  acquire.  Government  and  private 
acquisition  professionals  working  with  large  organizations  are  under  increased  pressure  to  cost- 
justify  training  system  acquisition  decisions,  yet  in  the  absence  of  a  complete  representation  of 
the  cost,  schedule  and  training  performance  analysis  model,  such  decisions  may  be  suboptimal. 
Specifically,  the  EVALS-R  system  addresses  common  pain  points  or  challenges  existing  in  this 
work  space,  as  shown  below  in  Table  4. 
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Table  4.  Benefits  of  EVALS-R 


Current  training  system  EVALS-R 

selection  decision  support 
challenges 


•  . .  .depend  on  decentralized 
spreadsheet  based  analysis 
methods  focused  on  technical 
capability  validation. 

•  . .  .are  technology  centered,  focus 
heavily  on  highest  fidelity 
solution  for  lowest  cost  and 
instead  of  ensuring  the  system 
requirements  justify  their  cost. 

•  . .  .focus  heavily  on  traditional 
ROI  metrics  which  limit  the 
ability  to  quantify  “soft”  benefits 
such  as  training  effectiveness. 

•  . .  .limit  analysis  of  system 
requirements  to  those  which 
impact  training  environment 
physical  fidelity. 

•  ..  .are  not  practical  for  use  under 
typical  end  user  constraints  (e.g., 
requires  extensive  expertise, 
time,  and  does  not  present  the 
results  in  an  easy  to  use  format).. 


•  . .  .utilizes  network  capable  technology 
to  allow  multidisciplinary  teams  work 
and  manage  ROTI  assessments. 

•  . .  .utilizes  a  training-needs  centered 
approach  to  determine  system 
requirements  necessary  to  train 
specific  skill  sets. 

•  . .  .extends  ROTI  to  facilitate  weighing 
the  training  benefit  (training 
effectiveness)  of  the  system  against 
cost  and  schedule  impacts 

•  . .  .expands  requirements  analysis  to 
incorporate  functional  fidelity  and 
psychological  fidelity. 

•  ...employs  wizard-type  interface 
designs  to  guide  users  and  presents 
results  in  a  visual  and  interactive 
format  which  encourages  iterative  cost 
benefit  trade-off  analyses 


The  target  markets  for  the  EVALS-R  system  are  government  training  acquisition  offices  and 
government  laboratories  given  the  budgetary  pressures  being  placed  on  all  agencies,  particularly 
the  Department  of  Defense  Services  which  rely  heavily  on  training.  As  budgetary  pressures 
increase,  the  need  to  gain  efficiencies  on  all  fronts  also  increases,  and  as  a  result,  more  efficient 
and  effective  forms  of  training  are  required.  Simulation-based  training  systems  have  been 
highlighted  as  a  means  by  which  greater  efficiencies  and  training  effectiveness  may  be  achieved 
while  at  the  same  time  reducing  or  eliminating  risks.  Nonetheless,  the  quality  of  these 
simulations  is  hard  to  quantify,  particularly  when  evaluating  their  impact  on  training.  This  is 
because  training  effectiveness  in  itself  is  challenging  to  quantify,  not  only  due  to  the  resource¬ 
intensive  nature  of  such  evaluations  but  also  because  of  the  multiple  methods  which  may  be 
utilized.  This  compounds  the  challenge  of  accurately  and  objectively  calculating  meaningful 
ROTI  analyses  that  not  only  focus  on  costs  but  also  on  time  and  training  performance.  EVALS-R 
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provides  a  solution  to  this  challenge  by  providing  a  single  tool  to  carry  out  all  necessary  sub¬ 
processes  for  calculating  meaningful  and  objective  ROTI  analyses. 


4.4  Technology  Transfer 

In  order  to  ensure  that  the  EVALS-R  system  is  ready  to  be  acquired  by  a  customer,  extensive 
testing  was  conducted,  not  only  throughout  the  development  process,  but  regression  testing  was 
also  conducted  at  the  end  of  the  final  development  version.  All  major  defects  and  usability  issues 
were  fixed,  and  the  system  was  optimized  to  run  as  efficiently  as  possible.  In  addition,  a  user 
manual  and  installation  materials  were  developed  for  delivery  to  customer. 

Throughout  Phase  II,  efforts  to  identify  and  transition  the  technology  to  customers  have  been 
conducted.  Discussions  with  representatives  from  government  agencies  such  as  PM  TRADE,  PM 
TRASYS,  and  NAWCTSD  have  been  pursued  following  demonstrations  of  the  technology.  In 
addition,  private  and  academic  organizations  have  been  identified  as  additional  potential 
transition  customers.  Discussions  with  a  Research  Hospital  and  Large  DOD  contractor  have  also 
been  pursued  in  order  to  evaluate  the  technology  and  identify  further  uses.  However,  though 
feedback  was  positive  and  there  was  interest  in  acquisition  in  some  cases,  nothing  has  yet  been 
able  to  be  worked  out.  Therefore,  transition  leads  continue  to  be  pursued. 

Further,  a  Phase  II  effort  funded  by  the  Air  Force  is  currently  underway,  involving  development 
of  a  software  system  to  support  a  user  in  conducting  different  types  of  Training  Effectiveness 
Evaluations  (TEE;  e.g.  based  on  Fidelity  Support,  Subjective  Reactions,  and  Learning  Gains). 
Design  Interactive  Inc.  developed  this  system’s  design  to  readily  integrate  with  EVALS-R  so  that 
the  training  system  performance  data  resulting  from  empirical  TEEs  can  be  pulled  into  the  ROTI 
analysis  to  ensure  that  it  includes  the  most  recent  and  comprehensive  training  effectiveness  data 
for  each  training  system  being  compared.  The  combination  of  these  two  efforts  allows  EVALS-R 
users  to  conduct  additional  empirical  TEEs,  beyond  the  fidelity  evaluation  capability  that  is 
currently  available,  increasing  the  accuracy  of  the  ROTI  results. 

5.0  CONCLUSIONS 

This  Phase  II  effort  resulted  in  the  development  and  validation  of  the  Evaluation  of  Value  Added 
in  Learning  Systems-RITE-MT  (EVALS-R)  tool,  a  decision  support  tool  for  the  conduction  of 
Return  on  Training  Investment  analysis  by  use  by  training  development  and  acquisition 
professionals.  EVALS-R  provides  these  practitioners  with  the  analytic  capabilities  to  assess  a 
training  system’s  ability  to  meet  training  goals,  within  the  budgetary  and  schedule  constraints  set 
forth  by  the  organization,  and  facilitates  return  on  investment  tradeoff  analyses.  As  a  result 
training  system  design/development  and  acquisition  decisions  made  with  support  of  EVALS-R 
will  lead  to  increased  organizational  training  and  readiness. 
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7.0  APPENDIX  A  -  Productivity 

Conference  presentation  (Contributed): 

Carroll,  M.,  Champney,  R.,  Padron,  C.,  Wang,  X.,  Haddad,  D.  Predictive  Approach  for 
Assessment  of  Return  on  Training  Investment.  International  Test  and  Evaluation  Symposium; 
2014  November  12-15;  Crystal  City,  Virginia. 
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8.0  APPENDIX  B  -  Award  Participants 

People  receiving  salary  support  from  the  EVALS-R  effort  include  Christina  K.  Padron,  Roberto 
K.  Champney,  Zachary  Huber,  Michelle  A.  Sinagra,  Jack  L.  Hart,  David  Mui,  Denis  Oliva 
Ramos,  and  Shauwn  Rush. 
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LIST  OF  SYMBOLS,  ABBREVIATIONS,  AND  ACRONYMS 

EVALS-R  Evaluation  of  Value  Added  in  Learning  Systems 
GUI  Graphical  User  Interface 

RITE-MT  Return  on  Investment  Tool  for  Effective  Medical  Training 

ROTI  Return  on  Training  Investment 

SME  Subject  Matter  Expert 

TEE  Training  Effectiveness  Evaluation 

TNA  Training  Needs  Analysis 
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